Event-driven systems ingest data elements and, in response, perform a series of computations in order to generate events based at least in part upon the ingested data elements. Consumers of these event-driven systems may listen for the events in order to be informed and, in some instances, to respond based upon the events. As event-driven systems become larger, the event-driven systems must generally be capable of receiving a large number of data elements and, in turn, performing a very large number of computations in order to generate the resulting events. Consequently, an event-driven system may have a substantial processing load and, as such, a substantial investment may need to be made in processing resources in order to generate the events in a timely manner.
Some event-driven systems have performed batch-based computations on all incoming data elements. By performing batch-based computations on all incoming data elements, these event-driven systems may reduce, at least somewhat, the processing resources that are required to generate the events, but do so at the cost of latency in the generation of the events. As such, consumers are not provided with real-time or even very timely generation of the events in at least some instances of event-driven systems that rely upon batch-based computation. Other event-driven systems operate in real-time or near real-time. These event-driven systems receive the data elements and generate the events with near real-time calculations. Although the events are generated in a more timely manner so as to satisfy the consumer, the performance of the multitude of calculations required to generate the events in real-time requires an investment in substantially greater processing resources. In particular, real-time event-driven systems, such as real-time streaming systems, may experience substantial compute peaks (the maximum consumption of processing resources by an event-driven system) in response to the receipt of a substantial amount of data elements at approximately the same time or in the same relatively small time period which, in turn, causes a need to perform a substantial number of calculations at approximately the same time in order to generate the resulting events in a timely manner. This compute peak may therefore require real-time event-driven systems to invest even more greatly in processing resources.
As such, an event-driven system for the generation of events based upon the receipt of data elements provides valuable information to consumers of the event, but suffers from either increased latency as an increasing number of computations must be performed to generate an increased number of events or increased costs as more processing resources are required to satisfy the greater compute peaks.